Systems, apparatuses and methods for controlling prosthetic devices by gestures and other modalities

ABSTRACT

Systems and methods for manual gesture recognition to control prosthetic devices. Low encumbrance systems utilizing glove-based recognition to control prosthetic devices. Prosthetic control systems and methods are also provided utilizing elements for application on the user&#39;s fingernails.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/488,500, filed Apr. 16, 2017, which itself claims priority from U.S.Provisional Application Ser. No. 62/323,592, filed Apr. 15, 2016, theentire disclosure of which is incorporated herein by this reference.

GOVERNMENT INTEREST

This invention was made with government support under Grant No. # NRIIIS-1208623 awarded by National Science Foundation. The government hascertain rights in the invention.

TECHNICAL FIELD

This disclosure relates generally to systems and methods in the field ofwearable electronics and human control interfaces. More specifically,the disclosure relates to techniques for recognizing signals to controlpowered prosthetic devices providing extended freedom of motion andoptional modalities of operation.

BACKGROUND

The state of the art for commercial robotic prosthetic systems is to usea pair of electromyography (EMG), also referred to as surfaceelectromyography (SEMG), sensors on the residual limb. These sensorsdetect the electrical activity of the muscles in the user's remaininglimb during muscle activation. For prosthetic hand devices, grasppatterns are typically selected by scrolling through “menus” byco-contracting different muscles (e.g., biceps and triceps) and puttingthe prosthetic device in different operating modes. Some systems alsoallow grasp selection via apps running on a smart phone.

Conventional prosthetic systems allow users to regain some lostfunctionality, but in a limited way. For example, prosthetic handdevices typically allow the user limited “open” or “close”functionality. While current powered prosthetic devices boast multipledegrees of freedom and dexterity levels that allow a user to perform anumber of operations and movements, interaction with these devices isstill limited, and limiting, outside of the clinical setting. EMG dataacquisition is highly variable and subject to noise due to lineardistances along the surface of the skin above the sensed muscle.Perspiration and user fatigue can also cause degraded deviceperformance.

The use of glove devices as an input interface has been demonstratedextensively in virtual reality environments. In the last few decades thegaming industry has used glove devices for glove-based input. Althoughsome research has been done relating to the use of glove devices inother fields, none has exclusively focused on the control of poweredprosthetic devices.

Accordingly, a need remains for improved low encumbrance systems andmethods for controlling powered prosthetic devices.

SUMMARY

Embodiments of the present invention provides systems, apparatuses, andmethods for recognizing (e.g., finger or hand) gestures and controllingexternal (e.g., prosthetic hand) devices based on the recognition of thegestures.

According to a first aspect of the invention, there is provided a systemcomprising a signal element configured for disposal on a first finger ofa user; and a detection element configured for disposal on a secondfinger of a user, and configured to detect a signal generated by thesignal element, wherein the signal indicates that a gesture has beenperformed using the user's fingers.

According to a second aspect of the invention, there is provided asystem comprising a plurality of conductive thimbles configured fordisposal on a user's fingers, the user's fingers being fingers of afirst arm of the user, wherein the conductive thimbles are configured tooutput a signal representing a gesture performed with the user'sfingers.

According to a third aspect of the invention, there is provided asystem, comprising a glove configured to be worn on a user's first hand,the user's first hand being a hand of a first arm of the user, whereinthe glove is configured to output a signal indicating a gestureperformed with the user's first hand.

According to a fourth aspect of the invention, there is provided amethod comprising outputting a signal corresponding to a gesture; andcontrolling a prosthetic device based on the outputted signal, wherebythe prosthetic device is controlled based on the gesture.

Other systems, apparatuses, and methods are also provided.

BRIEF DESCRIPTION OF THE DRAWINGS

The following figures form part of the present specification and areincluded to further demonstrate certain aspects of the present claimedsubject matter, and should not be used to limit or define the presentclaimed subject matter. The present claimed subject matter may be betterunderstood by reference to one or more of these drawings in combinationwith the description of embodiments presented herein. Consequently, amore complete understanding of the present embodiments and furtherfeatures and advantages thereof may be acquired by referring to thefollowing description taken in conjunction with the accompanyingdrawings, in which like reference numerals may identify like elements,wherein:

FIG. 1 is a schematic drawing of a manual gesture recognition andcontrol system disposed on a user's hand, according to some embodiments;

FIG. 2 is a schematic drawing of a pinch gesture performed with agesture recognition and control system disposed on a user's hand,according to some embodiments;

FIG. 3 is a schematic drawing of another manual gesture recognition andcontrol system disposed on a user's hand, according to some embodiments;

FIG. 4 is a schematic drawing of another manual gesture recognition andcontrol system disposed on a user's hand, according to some embodiments;

FIG. 5 is a schematic drawing of another manual gesture recognition andcontrol system disposed on a user's hand, according to some embodiments;

FIG. 6 is a schematic drawing of a pinch gesture performed with agesture recognition and control system disposed on a user's hand,according to some embodiments;

FIG. 7 is a schematic drawing of another pinch gesture performed with agesture recognition and control system disposed on a user's hand,according to some embodiments;

FIGS. 8A-8I are images of a gesture recognition and control system usingconductive thimbles disposed on a user's hands, according to someembodiments;

FIGS. 9A-9C are images of a glove-based recognition system, according tosome embodiments, and FIGS. 9D-9G are images of hand gestures performedwith the glove-based recognition system, according to some embodiments;

FIG. 10 is a schematic diagram of an operating hierarchy for a gesturerecognition and control system, applicable to multiple types of suchsystems, according to some embodiments;

FIG. 11 is an image of a sleeve (socket) device, which may be used in asystem for controlling an external device, according to someembodiments;

FIG. 12 is an image of a gesture recognition and control system,according to some embodiments;

FIG. 13 is an image of a gesture recognition and control system disposedon a user's hand, according to some embodiments; and

FIG. 14A-14D are images of a gesture recognition and control systemdisposed on a user's hand, according to some embodiments.

NOTATION AND NOMENCLATURE

Certain terms are used throughout the following description and claimsto refer to particular system components and configurations. As oneskilled in the art will appreciate, the same component may be referredto by different names. This document does not intend to distinguishbetween components that differ in name but not function. In thefollowing discussion and in the claims, the terms “including” and“comprising” are used in an open-ended fashion, and thus should beinterpreted to mean “including, but not limited to . . . ”

Various terms are now explained, although their meanings may be moreeasily grasped upon reading of the full detailed description set forthbelow in conjunction with the explanation given here. The term “pinchgesture,” as used herein, is explained as follows. A pinch gesture is amanual gesture that is performed by bringing together or touchingtogether (e.g., to each other, to one or more other finger(s), or toanother part of the hand) one or more fingers on the hand being used forinput. Such a gesture can be used to actuate an external device, such asa robotic prosthetic limb (e.g., a hand worn on the user's other arm), amusic player, a machine, etc. Many different gestures can be defined andused to cause respective different operations to be performed on anexternal device. For example, a finger pad of one finger may be touchedto or brought together with a finger pad of another finger, a finger padof one finger may be touched to or brought together with a fingernail ofanother finger, finger pads of two fingers may be touched to or broughttogether with a finger pad of another finger, and so on. In some aspectsof the invention, pinch gestures include gestures performed by bringingtogether or touching together two or more fingers, or touching thefingers to various parts of the hand, such as different locations on thepalm. Thus, a pinch gesture may but need not involve pinching in theordinary sense of the term. As will be understood by one of ordinaryskill in the art, embodiments described herein may apply to finger andhand (manual) gestures generally. The speed and frequency of thegestures (such as a double-tap, analogous to a mouse double-click or alonger-than-normal gesture, analogous to a long-press on a touch screen)can also be used to distinguish user intent. As used herein, the terms“recognition” and “tracking” may be interchanged.

It should also be noted that in the instant disclosure the term “finger”may refer to any finger, including the thumb. It is also noted that inthe instant disclosure the term “light” may refer to but is not limitedto the visible spectrum. The instant disclosure makes reference todetecting various characteristics of a signal, e.g., a frequency; acolor; an RFID tag ID; a signal intensity, strength, or amplitude, or achange in one of these; a timing; a resonant frequency of a circuittransmitting or generating the signal; and a frequency or pulse pattern.All of these characteristics and the like may be referred to as“characteristics” of the signal. The term “timing” of the signal mayrefer to any time aspect of the signal detected, such as a time ofreception or transmission of the signal (e.g., relative to time oftransmission of an interrogation signal triggering the signal as aresponse signal from the signal element), a characteristic response timefor a given finger, or any other time characteristic of the signaldiscussed herein.

The instant disclosure uses the term “characteristics” of a gesture torefer to various characteristics of a gesture, such as a particularfinger or fingers used, a particular position of one or more fingers, aparticular pose or grasp pattern/grip pattern of one or more fingers, aparticular movement performed by one or more fingers, and a state ofcontact or non-contact of one or more fingers with another one or morefingers or with another part of the hand. By “a particular finger orfingers used” is meant an indication or identification of whichparticular finger(s) are being used in the performance of the gesture.Further, the instant disclosure refers to a controller controlling aprosthetic device based on a detected signal (corresponding to agesture). In some cases, such control may be performed directly by thecontroller, while in other cases such control may be performedindirectly by the controller, that is, via one or more intermediatedevices, such as another computer device. The instant disclosure alsorefers to a signal representing muscle force or pressure; this locutionmay refer also to a signal representing a change in muscle force. Andmore generally, as will be understood by one of ordinary skill in theart upon reading the instant disclosure, in some cases the disclosuremay refer to detecting a quantity as a shorthand for detecting eitherthe quantity or a change in the quantity (the quantity being a signal ora characteristic of a signal being detected, such as the frequency oramplitude of an electrical signal, light, etc.). Finally, the instantdisclosure may refer to causing a prosthetic device to perform an actionas a shorthand for causing a prosthetic device to perform an action,change position or move in a certain manner (e.g., a mannercorresponding to a detected signal/recognized gesture); in other words,in this context, ‘performing an action’ may but need not refer tochanging position or moving in a certain manner.

DETAILED DESCRIPTION

The foregoing description of the figures is provided for the convenienceof the reader. It should be understood, however, that the embodimentsare not limited to the precise arrangements and configurations shown inthe figures. Also, the figures are not necessarily drawn to scale, andcertain features may be shown exaggerated in scale or in generalized orschematic form, in the interest of clarity and conciseness. The same orsimilar parts may be marked with the same or similar reference numerals.

While various embodiments are described herein, it should be appreciatedthat the present invention encompasses many inventive concepts that maybe embodied in a wide variety of contexts. The following detaileddescription of exemplary embodiments, read in conjunction with theaccompanying drawings, is merely illustrative and is not to be taken aslimiting the scope of the invention, as it would be impossible orimpractical to include all of the possible embodiments and contexts ofthe invention in this disclosure. Upon reading this disclosure, manyalternative embodiments of the present invention will be apparent topersons of ordinary skill in the art. The scope of the invention isdefined by the appended claims and equivalents thereof.

Illustrative embodiments of the invention are described below. In theinterest of clarity, not all features of an actual implementation aredescribed in this specification. In the development of any such actualembodiment, numerous implementation-specific decisions may need to bemade to achieve the design-specific goals, which may vary from oneimplementation to another. It will be appreciated that such adevelopment effort, while possibly complex and time-consuming, wouldnevertheless be a routine undertaking for persons of ordinary skill inthe art having the benefit of this disclosure.

FIG. 1 illustrates a system 100 for recognizing pinch gestures andcontrolling a prosthetic device based on the recognized gestures,according to an embodiment. (A system for recognizing gestures andcontrolling a prosthetic device based on the recognized gestures may bereferred to as a gesture recognition and control system.) In thisembodiment, all recognition components of system 100 are disposed on theuser's fingernails 101. As illustrated in FIG. 1, a signal element 102comprising an LED 104 is disposed on the user's fingernail 101. Thesignal element 102 is configured with one or more LEDs 104 and LEDdriver circuitry 106 (including batteries). A detection element 108 isdisposed on the user's thumbnail 103. The LED signal element 102 ispositioned facing the fingertip or fingernail, to transmit light suchthat the detection element 108 (in this case on the thumb) detects thelight when a pinch gesture is performed as shown in FIG. 2. (FIG. 2intends to show a case of bringing the finger pad to touch the thumbpad.) In other embodiments, it's also possible to point a light detectoraway from the fingernail, allowing for the performance of a second typeof pinch gesture by touching the finger pad to the thumbnail, instead oftouching the finger pad to the thumb pad. Although FIG. 1 shows only onefinger provided with a signal element 102, all four fingers can beequipped with individual signal elements 102 (see, e.g., the embodimentof FIG. 3). In other embodiments, the detection element 108 may bedisposed on another finger, instead of the thumb, with the signalelement(s) 102 on the other finger(s) appropriately positioned forsignal alignment.

For embodiments implemented with signal elements 102 disposed on morethan one finger, the individual LED signal elements 102 can be modulatedor pulsed at a frequency specific to each finger. The color of the LED104 may also be chosen so that it can transmit through 1-3 cm of tissue(generally red or infrared) at a reasonable intensity. The user performsa gesture by bringing together or touching one or more fingers equippedwith a signal element 102 to the finger containing the detection element108. Light from the LED signal element 102 traverses the fingers and isdetected by the detection element 108.

The LED-based detection element 108 comprises a light detector 110(e.g., a photodiode, photoresistor) and microcontroller circuitry 112(including a battery) to detect which finger or fingers are in closeproximity or in contact with the finger on which the detection element108 is mounted. The LED-based system 100 takes advantage of the factthat certain frequencies of light are transmitted through human tissue.Several methods can be used to distinguish between different signalelements 102/fingers, including but not limited to pulsing the signalelements 102 at different frequencies, choosing LEDs 104 or acombination of LEDs 104 on each finger of slightly different colors(infrared, red, yellow, etc.), or using a mechanism on each signalelement 102 which is triggered by a light pulse or a Radio Frequency(RF) pulse being emitted by the device on the approaching finger, whichthen responds by turning on the LED signal element 102. The detectionelement 108 is also configured with an antenna 114 to receive the signaltransmitted by the signal element 102. The microcontroller 112 producesan output signal based on the signal received by the detection element108. The output signal may be used, directly or indirectly, e.g. via acomputer, to control an external device, such as a prosthetic device.The user may have one healthy arm and one disabled arm, and the user mayuse the healthy arm to wear the system 100 components (e.g., signalelements and detection elements) and perform the manual gestures, andthe prosthetic device may be disposed on the user's disabled arm and maybe controlled, via the system 100, by the gestures performed by thehealthy arm. The control may involve causing the prosthetic device toperform an action, such as to take a certain position or pose, or movein a certain manner. The specific action performed, position/pose taken,or manner of movement would correspond to the specific gestureperformed. Further description of this control of the prosthetic devicebased on the performed gestures is given elsewhere in this disclosure,and the description given here and elsewhere may apply to variousembodiments even if that description is not provided together with thoseembodiments and not explicitly stated to be applicable to thoseembodiments, as will be understood by one of ordinary skill in the art.(More generally, it is noted that, for the sake of brevity, for someembodiments, not all elements of a gesture recognition and controlsystem are described. For example, in some embodiments, description ofthe control of the external, e.g., prosthetic, device, includingdescription of the microcontroller for producing output for performingthat control based on the signal received by the detection element, isomitted. One of ordinary skill in the art will understand that, wherethe disclosure omits description of such functionality or structuralelements for a given embodiment, the teachings pertaining to thatfunctionality or structural element as included for other embodimentsmay be applicable to the embodiment for which the description of thatelement was omitted.)

FIG. 3 illustrates another gesture recognition and control system 300,according to an embodiment. In this embodiment, inductively coupledRadio Frequency Identification (RFID) tags 304 are disposed on theuser's fingernails 301. The detection element 308 is a detachable ringfixture on the user's thumb 303, configured with a miniature RFIDantenna 314, with associated electronics 310 to support tag reading, aswell as electronics 312 to transmit information output, and powerelectronics. The signal elements 302 are the miniature RFID tags 304disposed at the respective tips of the four fingers so that thedetection element 308 on the thumb can discriminate which finger isnearest to the antenna 314 for subsequent pinch gesture recognition.Standard passive RFID tags (whether magnetically or electricallycoupled) have individual unique IDs, in turn allowing for uniqueidentification of the fingers without additional complexity. In someembodiments, magnetically coupled RFID tags may be optimal consideringthe short reading distances employed and the overall low power operationof such readers.

Various arrangements of antennas 314 and corresponding electronics 310,312 are possible. For example, FIG. 3 shows a two-piece device includingan antenna 314 disposed on the thumbnail and connected to a ring mountedon the thumb that contains the RF electronics 310 for interrogating theRFID tags 304 of the signal elements 302 and receiving responses fromthose RFID tags 304, and the microcontroller circuitry 312 forcontrolling an external device based on the received responses. Theantenna 314 can be disposed on the thumbnail using any suitable means(e.g., direct adherence via an adhesive, via an elastic band wrappingaround the fingertip to hold the antenna 314 in place for easy removal,etc.). Such implementations allow for positioning of the antenna 314parallel the thumb. In other arrangements, the detection element 308,including both the antenna 314 and associated electronics 310 and 312,may be housed entirely within in the ring on the thumb (e.g., not on thethumbnail). The antenna 314 incorporated on the ring body may beperpendicular to the finger. In other arrangements, the antenna 314 mayhave any of various configurations and orientations/positions.Embodiments implemented with a detection element 308 having only oneantenna/reader may not be feasible for triangulation and/or localizationof the signal element tags 304. However, in the general case of RFcircuitry, signal element tag identification is feasible given thereading proximity of the disclosed implementations. Additionally, planardistance may be measureable based on measured signal intensity orthrough adjustment of interrogating frequency amplitudes to obtaingeneral distance information.

FIG. 4 illustrates another gesture recognition and control system 400,according to an embodiment. In this embodiment, multiple signal elements402 (each containing an RFID tag 404) and multiple detection elementsare disposed on the user's hand. As shown in FIG. 4, a multiple antennaconfiguration is disclosed, including a plurality of antennas(collectively referred to as 414) on the user's hand, namely, a thumbantenna 414 a located on the thumb, an upper palm antenna 414 b locatedon the palm toward the fingers, and a lower palm antenna 414 c locatedtoward the wrist. Thumb antenna 414 a may be disposed either on thethumbnail 403 (first orientation) or around the thumb (secondorientation); in some arrangements, both of these thumb antennas 414 amay be employed. With this configuration, one can spatially localize andtriangulate the position of the signal element tag 404 in 3D space forfurther improved gesture recognition. By measuring signal intensity ofthe RFID tag 404 as acquired by a combination of the antennas and/or orby measuring the time between interrogation and data transmission (e.g.,with synchronized or asynchronized electronics) between the antennas andtags, one can obtain a position of the RFID tag 404 in two dimensions(2D) relative to the plane of the palm as well as in three dimensions(3D). This can be accomplished in several ways. (For convenience, thecontroller and electronics, etc. associated with the antennas 414 arenot shown in FIG. 4.)

One way is by triangulation based on signal intensity. This is based onthe assumption that the attenuation of the signal by the surroundingfingers is negligible. For any RFID tag being queried, more than onesensor can be used to read the signal. The amplitude of the signal ateach detector can be measured. The strength of the signal willcorrespond to the distance of the sensor to the RFID tag.

Another way is by triangulation based on time of flight. Withsufficiently fast detectors, the time of arrival of the signal from theRFID tag can be measured accurately. Since the signal will arrive at adetector which is further away later than at a detector which is closeby, the distance of the RFID tag to each sensor can be calculated. Thiswill yield an approximation of the location.

In some embodiments, the two triangulation techniques described abovecan be combined with a model of the hand, adding geometrical constraintsto the RFID tag location estimation. In another way, triangulation basedon signal intensity can be combined with machine learning approaches to“train” a machine learning system (such as an Artificial Neural Network)with different “ground truth” hand poses and signal intensities asinputs. The system would learn to associate a combination of signalintensities for each RFID tag with a given hand pose. With asufficiently large number of such training inputs, the system can learnto estimate hand poses without explicit knowledge of hand geometry orsignal attenuation. In yet another way, triangulation based on signalintensity can be combined with Radio Frequency attenuation andpropagation modeling, similar to approaches used in antenna designsystems. Conventional commercial simulators which can model RFpropagation may be used. Since the general shape of the hand and theposes it can achieve are well known, measured RF attenuation at eachsignal can be compared to what is theoretically expected for a varietyof different poses, resulting in a pose estimate.

Embodiments along the lines of system 400 illustrated in FIG. 4 allowfor the detection of additional gestures such as gestures used in signlanguage. Such embodiments may also be used to directly tele-operate ahand-like robotic manipulator, such as a prosthetic arm mountedcontra-laterally, or an industrial robot. The triangulation techniquesdescribed above can also be implemented with other embodiments disclosedherein, including embodiments configured for light or sound signaling.

FIG. 5 illustrates another gesture recognition and control system 500.This embodiment presents an alternative scheme for radio frequencyinterrogation for determining position and identity of the correspondingfinger on which the circuit is placed. For signal elements 502, thisimplementation uses individually tuned LC resonant tank circuits 504with an inductor (L) and a capacitor (C) placed in series. The circuit504 is used as an electrical resonator, storing energy oscillating atthe circuit's resonant frequency. Depending upon the pre-calculatedspecifications of the circuit 504, the circuit 504 can ‘ring’ orresonate when interrogated with the established resonant frequency. Someembodiments encompass the use of separately tuned signal elements 502(one for each finger), and a detection element 508 with a transmitterand receiver, or a transceiver, such as a transmit/receive coil 514.Different methods for transceiving, or sending and receiving signals,may be used. One exemplary method comprises a voltage controlledoscillator to generate frequencies necessary to ‘ring’ or resonate theLC circuits 504. A software defined radio (a simplistic radio frequency(RF) spectrum analyzer) is used to read the signal being sent back bythe LC circuit 504. For example, conventional software applicationsproviding real-time signal spectral analysis may be used to determinethe frequency of the signals output by the LC circuit 504. While FIG. 5shows multiple antennas 514, some embodiments may be implemented withother antenna configurations and additional software defined radios.Various arrangements along the lines of FIG. 5 may allow fortriangulation in 2D and 3D recognition. Other embodiments may beimplemented incorporating detection elements 508 (which are also LCcircuits) that are tuned to the resonant frequency of the signal elementcircuits 504 mounted on the fingers, using simple circuitry to measurethe produced signals.

FIG. 6 illustrates a pinch gesture performed using another gesturerecognition and control system 600. System 600 includes a detectionelement 608 with an RF transmitter 609 and a light detector 610 mountedon one finger, and a signal element 602 comprising a receiving RFantenna 607 and an LED 604 (with LED driver 606) facing the fingernail601 mounted on another finger. In some embodiments one or more of theother fingers are each configured with individual signal elements 602,and each LED driver 606 can be tuned to respond to a different RFfrequency or modulated RF signal. The detection element 608 cancontinually alternate between sending an RF pulse which activates therespective LED 604 on each successive finger. If a particular finger isin range, the LED driver circuit 606 is activated and responds with asingle light pulse, which travels through the sending finger and thereceiving finger, to be detected by the light detector 610 on thedetection element 608. This embodiment has the advantage of savingbattery life on the LED driver circuits 606 mounted on each fingernail601, allowing them to be less cumbersome and lighter (e.g., to beconfigured as cosmetic “snap on” fingernails).

FIG. 7 illustrates a pinch gesture performed using another gesturerecognition and control system 700. In system 700, a detection element708 with an LED 709 and a light detector 710 is disposed on one finger,and a signal element 702 with LED driver circuitry 706 comprising alight detector 707 and an LED 704 facing the fingernail is disposed onanother finger. In this embodiment, the LED driver 706 can be tuned torespond to different light pulse patterns and frequencies. When thedetection element LED 709 outputs light, it emits the light at a certainfrequency, or in a specific frequency or pulse pattern. The LED driver706 can be configured (in cooperation with the signal element lightdetector 707) to detect a unique frequency or frequency/pulse pattern ofthe light emitted by the detection element LED 709. In an embodimentwherein the other fingers are each configured with individual signalelements 702, the detection apparatus 708 can continually alternatebetween sending light pulses which would activate the signal element LED704 on each respective finger. If a particular finger is in range, thesignal element LED driver circuit 704 is activated and responds with asingle light pulse, which travels through the sending finger and thereceiving finger, to be detected by the light detector 710 on thedetection element 708. This embodiment also has the advantage of savingbattery life on the LED driver circuits 706 mounted on each fingernail,allowing them to be less cumbersome and lighter.

Various embodiments described herein may be realized using a plastic“snap on” fingernail configured with miniaturized components, e.g., alight detector (e.g., facing the fingernail), a surface mount miniatureLED (e.g., facing the fingernail), a miniaturized LED driver, an antennato relay output to an external receiver, a battery (such as a watchbattery), and the like. The snap on fingernail may be produced using anysuitable material as known in the art, with the components mounted onthe surface using a suitable adhesive.

As noted above, using the gesture recognition and control systemsdescribed herein, gestures can be used to actuate an external device,such as a prosthetic limb. Embodiments described below also includegesture tracking and control systems that are worn on a user's sound(healthy, normal) hand in order to detect different grasp patterns orposes, which are used to control a powered prosthetic device.

FIGS. 8A-8I illustrate a gesture tracking and control system 800,showing, inter alia, different gestures. System 800 comprises conductivethimbles 820, one worn on each finger, which are used to control aprosthetic device, such as a prosthetic limb. The thimbles 820 areformed with a conductive outer surface (e.g., a conductive metallicfilm). In some embodiments each of the thimbles 820 may be formed as aconductive metal ‘cup’ disposed onto an insulating material (e.g., aplastic, foam, or rubber layer with a cotton lining) configured to fitthe user's fingers. The thimbles 820 are electrically coupled to amicrocontroller board 812 (further described below). The electricalconnections may be made via wires 821 coupled to the thimbles 820 usingany suitable conventional connection means as known in the art. Contactbetween one or more fingers (each connected to an analog input on themicrocontroller) and the thumb (connected to the +5V output of themicrocontroller board) is interpreted as a gesture. Each of FIGS. 8C-8Iillustrates a different gesture. Each gesture corresponds to a specificgrasp pattern, not necessarily mirroring the pose of the gesture. Thatis, the gesture serves as an instruction to a controller to cause theprosthetic device to take on a specific grasp pattern corresponding tothe specific gesture. Grasp pattern activation was tested (including theselection of “open” and “closed” forms of a given pattern) both withdirect gestures, without relying on an EMG based input. FIGS. 8A and 8Bdo not illustrate gestures, but are provided to give a full view of thesystem components on all the fingers, on both sides of the hand.

FIGS. 9A-9G illustrate a gesture tracking and control system 900. System900 comprises a conductive glove 930 and thread 931 (the “conductiveglove”) to be worn on the healthy hand. The conductive glove 930 leavesthe healthy fingers free for the performance of non-system-relatedtasks. The conductive glove 930 is suitable for being worn for extendedperiods of time. In one embodiment, the conductive glove 930 wasconstructed from a fingerless weightlifting glove by embroideringconductive thread touch pads 931 in the palm of the glove 930 andconnecting them to snap connectors 932 sewn on the dorsal side of theglove 930. The conductive thread lines (not shown) going from theembroidered pads 931 to the snap connectors 932 were insulated withfabric affixed to the inside of the glove 930. (The conductive threadtouch pads 931 are shown in FIGS. 9B and 9C, and the snap connectors 932are shown in FIG. 9A.)

FIG. 10 illustrates an operating hierarchy for systems 800 and 900. Aswith the conductive thimble system 800, so too the conductive glovesystem 900 entails a microcontroller 912. Each of microcontroller 812and microcontroller 912 may be, e.g., an Arduino® microcontroller. Thethimble 820 or glove 930 is connected to the microcontroller board 812or 912, respectively, running the appropriate software. Themicrocontroller 812, 912 interprets the input from the thimbles 820 orthe glove 930 and creates commands, such as UNIX shell commands, whichare communicated to a computer 1040 (e.g., a Raspberry Pi™ computer)disposed on the prosthetic device 1050, and which, in turn, control theprosthetic device 1050 via a Controller Area Network (CAN) interface1060. Connections 1070 between microcontroller 812, 912, computer 1040,and interface 1060 may be made by any suitable connection means known inthe art, e.g., Universal Serial Bus (USB).

Returning to FIGS. 9A-9G, a contact pad (not shown; also referred to asa first electrically conductive pad) was embroidered on the inside ofthe glove 930 so that it makes contact with the skin and was connectedto the +5V terminal 933 of the microcontroller 912 board via anothersnap connector 932 (see FIG. 9A). In operation, the user touches one ofthe conductive pads 931 (also referred to as a second electricallyconductive pad) with a finger and completes the circuit causing a signalto be acquired by the A/D converter (of the microcontroller 932). Thesystem identifies touch gestures from each fingertip while notregistering any accidental gestures from objects that the user might bemanipulating. The skin contact with the +5V terminal 933 may be furtherenhanced by connecting the terminal 933 to a gel electrode 934 designedfor use with a Transcutaneous Electrical Neural Stimulator (TENS) system(Infiniti ELT 5050T) (FIG. 9B). The TENS electrode 934 is not strictlynecessary, but it improves the signal-to-noise ratio, largelyeliminating misclassified gestures. The output from the conductive pads931 was sampled with the built-in 10 bit A/D converter of the ATmega 328microcontroller 932 on the Arduino® board. FIGS. 9D-9G show examplegestures that can be performed with the conductive glove 930. A gestureis performed by touching one or more fingers to the conductive threadpad 931 located on the palm of the glove 930. FIG. 9D shows the gestureof touching the index finger 936 to the pad 931, FIG. 9E shows thegesture of touching the middle finger 937 to the pad 931, FIG. 9F showsthe gesture of touching the fourth finger 938 to the pad 931, and FIG.9E shows the gesture of touching the fifth finger 939 to the pad 931.Other gestures are possible, including touching the pad 931 with morethan one finger. Each gesture corresponds to a respective grasp pattern,i.e., the gesture is captured by microcontroller 912 and serves as aninstruction to control the prosthetic device on the other hand to takeon the grasp pattern.

In order to allow non-impaired volunteers to wear and test systemsdisclosed herein, a so-called healthy limb adapter was created. Aprosthetic device mounted on the end of the limb adapter was used totest the systems. A suitable prosthetic device (Touch Bionics™robo-limb) is produced by Touch Bionics Inc. (www.touchbionics.com). Thelimb adapter incorporates an Ottobock® Quick Connect ring (Otto Bock,Germany) attached via standard nuts and bolts via a custom 3D printedbracket attached to a plastic shell. The ring is disposed between theprosthetic hand and the shell. The shell was formed out of heat moldablethermoplastic (Worbla, USA) with a heat gun by using a plaster of Parisreplica of the subject's arm as a template. Power and data wires wererouted to the outside of the healthy limb adapter through a drilledhole. An armband (formed using Velcro®) is used to hold the circuitboards terminating a controller area network (CAN) cable and providingpower, batteries, and a Kvaser® Leaf (Kvaser, USA) USB to CAN adapter. ARaspberry Pi™ embedded computer, which interprets the signals from themicrocontroller to generate the CAN messages driving the prostheticdevice, is also affixed to the armband. A USB cable was used to connectthe tracking glove to the Raspberry Pi™ device on the healthy limbadapter.

In testing the embodiment implementations, for purposes of circuitdesign and software parameter specifications, the hand was treated as a1 to 2 MΩ resistor. By touching the conductive pad with a barefingertip, the circuit is completed, causing the A/D converter to see anincrease in voltage. A touch classification algorithm was developed inorder to classify touch events. For this, an algorithm using movingaverage and moving standard deviation was developed. The moving standarddeviation is a method that is useful in various applications where thesignal is noisy or the baseline is not stable. Further description ofthe classification touch algorithm, conductive thimbles, and conductiveglove embodiments of the invention is found in: Oguz Yetkin et al.,Control of a Powered Prosthetic Device via a Pinch Gesture Interface,University of Texas at Arlington, which is hereby incorporated herein byreference in its entirety. A copy of this paper is included as AppendixA in the provisional application to which the instant application claimspriority. The instant inventors are co-authors of this paper.

The systems disclosed herein provide interfaces to control externaldevices, such as powered prosthetics. For implementation of theembodiments by unilateral amputees, the system's devices are worn on theuser's sound hand to tele- operate the prosthetic device by mirroringthe user's gestures. Some embodiments are also configured to allow theuser to pose and lock the prosthetic device when independent handmovement of the healthy hand is desired. This feature allows the user to‘pause’ the prosthetic to allow different, independent, motion with thesound hand, i.e., when the user wishes to use the sound hand for somepurpose or task other than controlling the prosthetic device.

In this regard, other embodiments utilizing a glove implementation areprovided. In one embodiment, Sparkfun Electronics® 2.2 flexion sensors(Spark Fun Electronics, Inc.) are sewn onto the glove fingers. Amicrocontroller (e.g., Arduino® Uno microcontroller) is also affixed tothe glove using Velcro®. A sensor shield is also implemented with themicrocontroller. The shield contains circuitry to pre-amplify the outputfrom the piezoelectric sensors, along with buttons, potentiometers, andLED indicators for calibration without involving an external computer.The glove is also configured with a “pause” switch which can be actuatedby pressing against the body or an object. The switch temporarilydisables the mirroring functionality, allowing the user to performindependent manipulation with the sound hand. According to thisembodiment, for example, a user may perform a bilateral mirrored task(such as carrying a box), then pause the prosthetic device in the givenpose employed for the bilateral mirrored task, in order to perform adifferent manipulation (e.g., removing a bottle cap) with the soundhand.

The software and hardware architecture for this embodiment is similar tothe configuration represented in FIG. 10. The software utilizes threecomponents: a microcontroller, a standard UNIX BASH shell, and coderunning on a Raspberry Pi™ device which can open and close individualdigits via the command line. The software running on the microcontrollercomposes a UNIX command to set the position of each digit and send itover the USB cable to the Raspberry Pi™. The Raspberry Pi™ conveys thecommand received from the serial port to the BASH command interpreter,invoking the software with the set arguments. In some embodiments, thesoftware is configured to: issue the command to stop any movement thatmight be happening on the prosthetic device; issue the command to eachdigit to go to a desired/required position.

With further regard to this embodiment, an amplifier circuit may beemployed for each flexion sensor. Since the flexion sensors display ahigh resistance value (which results in a small voltage change when thesensor is bent), with each one having a different associated range,potentiometers are used in the circuit to allow for individualadjustment of each sensor's output.

For testing purposes, this embodiment was also configured and operatedusing the healthy limb adapter described above. Further description ofthese embodiments is found in: Oguz Yetkin et al., Control of a PoweredProsthetic Hand via a Tracked Glove, University of Texas Arlington,flyer; Oguz Yetkin et al., Evaluation of Prosthetic Control via Hand andGesture Tracking, University of Texas Arlington, paper; and in OguzYetkin et al., Control of a Powered Prosthetic Hand via a Tracked Glove,University of Texas Arlington, paper; all three of which are herebyincorporated herein by reference in their entirety. Copies of this flyerand both papers are included as Appendices B, C and D, respectively, inthe provisional application to which the instant application claimspriority. The instant inventors are co-authors of these papers andflyer.

Some embodiments are implemented to facilitate the control andfunctionality of a powered prosthetic device (e.g., by trans-radialamputees) through the use of intra-socket force readings, combined withthe pinch gesture glove or thimble embodiments disclosed herein. FIG. 11illustrates a socket utilized in certain embodiments.

More specifically, FIG. 11 shows a custom, prosthetic trans-radialamputee socket 1100 produced for an able-bodied man. An alginate moldwas created of the subject's dominant side lower arm, below the bicep,and a plaster cast was created from this mold. From this plaster cast, asocket 1100 was created by first wrapping the outside of the forearm inthe socket material, thermoplastic, and then layering subsequent layersinside of the socket 1100. The hand flexion and extension of the forearmwere identified on the subject, located on the anterior and posteriorsides of the forearm. These points were marked and correspondingpositions within the socket 1100 were identified. Above these positions,custom designed sensor housings 1180 were mounted, as shown in FIG. 11.These housings secure piezo-resistive force sensors 1181 (e.g.,FlexiForce® A201 piezo-resistors) for interaction detection. As the userflexes the arm muscles, the carpi flexor and carpi extensor muscleschange in volume. This volume change is detected as a force applied tothe inside of the socket 1100 and is detected by the piezo-resistivesensor 1181.

By implementing the piezo-resistive sensors in a prosthetic device(e.g., a prosthetic hand) such that skin contact is made, as the usercauses muscle deformation of the forearm during flexion and extensionactivities, the sensors detect the changes in force and pressure. Thistechnique is referred to as Force Myography (FMG). Embodiments disclosedherein include the use of such FMG interfaces used to controlfunctionality of powered prosthetic devices through the use ofintra-socket force readings. Embodiments disclosed herein also entailthe use of the Pinch Gesture devices disclosed herein, in combinationwith the use of intra-socket pressure sensors.

FIG. 12 shows an embodiment combining the intra-socket system and thepinch gesture glove system. This combination allows control andactivation via several modalities, including the use of gestures toselect between grasp patterns, use of hand tracking to tele-operate theprosthetic hand by mirroring the sound hand, activation of selected grippatterns via intra-socket pressure, and a method of turninggesture-based control on and off at will. As seen in FIG. 12, a healthylimb adapter, similar to that described above, was used to test thisembodiment. A Touch Bionics i-Limb™ ultra-powered prosthetic hand wasconfigured to demonstrate possible grip configurations. Each of sixdegrees of freedom are independently controllable, using an open-loopcontroller. The combination system of FIG. 12 includes, on the healthyhand, conductive glove 930 (for brevity, the components of glove system900 are not individually mentioned here), and, on the other arm, aprosthetic hand 1250, a socket 1100, and a Raspberry Pi™ computer 1040to control the prosthetic hand 1250 (for brevity, other elements on thelimb adapter, described above, are not individually mentioned here.)

A prosthetic device configured with the piezo-resistive force sensorsallows the user to control the opening, closing, and stopping of themotion of the powered prosthetic device. In some embodiments, userinteraction with this embodiment involves the selection of a grip-typevia an application stored on a mobile device or chosen by “scrollingthrough” pre-programmed grip patterns using a surface EMG as a scrollswitch. Opening and closing of the grip is completed via a predefinedinput through the surface EMG. Alternate methods of grip selection andcontrol algorithms for implementation of embodiments of the inventionare described in: Joe Sanford et al., A Novel EMG-Free ProstheticInterface System Using Intra-Socket Force Measurement and PinchGestures, University of Texas Arlington; Joe Sanford et al., ConcurrentSEMG and Force Myography Classification During times of ProstheticSocket Shift and User Fatigue, IEEE Journal of Transactions onBiomedical Engineering, February 2016; and Joe Sanford et al., SurfaceEMG and Intra-Socket Force Measurement to Control a Prosthetic Device,University of Texas Arlington; all three of which are herebyincorporated herein by reference in their entirety. Copies of thesepapers are included respectively in Appendices E, F and G of theprovisional application to which the instant application claimspriority. The instant inventors are co-authors of these papers.

Turning to FIG. 13, another embodiment of a fingernail-worn device fordetecting pinch gestures and communicating with the prosthetic device isshown. Similar to some other embodiments described herein, thisconfiguration detects the relative position of fingers to each other bymeasuring light transmitted via tissue. Fingernail mountedsignal/detection element or sensor holders 1305 for an LED 1304 and aphotodiode 1307 were modeled in SolidWorks® and 3D printed using afilament extruder. A high powered 639 nm LED 1304 and a broad spectrumphotodiode 1307 were friction fit into the sensor holder 1305. Both theLED 1304 and the sensor 1307 on each sensor holder 1305 are soldered tothin wires.

FIGS. 14A-14D illustrate a system 1400 using the fingernail-worn devicesof FIG. 13, the system being configured with one thumb sensor 1305 andtwo fingernail sensors 1305. The wiring 1421 is connected via DuPont®connectors to a wrist worn microcontroller 1412 board. Themicrocontroller 1412 board is constructed out of a Velcro® wristband, anArduino® Genuino microcontroller board, and a custom Arduino® shieldwith circuitry to drive the LEDs 1304 and read the input from thephotodiodes 1307 shown in FIG. 13. These embodiments take advantage ofthe fact that certain wavelengths of light travel through tissue. Sincea person's fingers generally have a similar index of refraction, thelight signal changes as the fingers come into physical contact and forma waveguide for the light, as well as when the fingers are breakingcontact. Light transmission between the fingers is greatly enhanced whenthe fingers are brought together physically. This change in signalenhancement provides for easier transmission from a fingernail to athumbnail, allowing light directed at a fingernail to be detected at asensor aimed at the thumbnail. While FIG. 14A shows the system as awhole most clearly, each of FIGS. 14B-14D shows a respective one ofthree different sample pinch gestures that can be performed with thissystem. In FIG. 14B, the pinch gesture is bringing the index finger 1436and the middle finger 1437 into contact with the thumb 1435; in FIG.14C, the pinch gesture is bringing the index finger 1436 into contactwith the thumb 1435; and in FIG. 14D, the pinch gesture is bringing themiddle finger 1437 into contact with the thumb 1435.

The embodiment of FIG. 13 was used to detect touch events. For thisdetection method of the invention, each detection cycle is broken intodetection windows of a specific duration (e.g., 200 ms). Each finger isassigned a characteristic response time (r1, r2, r3, r4). These timesare chosen to be smaller than the window duration (e.g., 25 ms, 40 ms,57 ms, 75 ms, etc.) but larger than pulse duration (e.g., 5 ms). Duringeach detection window, the LEDs are flashed in succession.Simultaneously, the microcontroller reads the light intensity on thephotodiode and looks for peak intensities corresponding to the responsetimes (r1, r2, etc.). Detection of such a peak is counted as a touchevent.

In some embodiments, the configuration depicted in FIGS. 13 and FIGS.14A-14D is modified slightly to allow the sensors on each finger to becontrolled by their own untethered microcontroller. These embodimentsare similar in operation to the time-based detection embodimentsdescribed in the preceding paragraph except that an interrogation pulseis sent from the thumb to the sensors mounted on each fingernail. If thesensor on the fingernail detects a light peak, it responds after acharacteristic response time (e.g., r1 for index finger, r2 for middlefinger, r3 for ring finger, r4 for little finger). Nominal values of,e.g., 25, 40, 57, and 75 ms can be chosen. The sensor on the thumbdetects the response and registers a touch event if one or more peaksare detected within response times corresponding to a finger. Theseembodiments have the advantage of allowing fingernail based sensors tobe completely untethered and manufactured inexpensively, as all theyneed to do is to respond to a light pulse. This furthermore increasesbattery life, since the fingernail mounted sensors only have to respondto a pulse if the fingers are in contact.

The time-domain based embodiments of FIG. 13 and FIGS. 14A-14D may beprone to noise, and an assumption can be made that the signal from theLED will be the brightest light pulse detected by the photodiode duringthe detection window. Thus a bright LED may be used to ensure betterperformance. Alternative embodiments may be implemented using a FastFourier transform (FFT) involving placing LEDs flashing in differentfrequencies on each fingernail, and detecting the light travelingthrough both fingers when the pinch gesture is performed. This greatlysimplifies the design of the fingernail mounted devices, as nomicrocontroller is needed on the four fingernails.

Signal processing for embodiments can be implemented via a Matlab® codewritten to take a vector of time domain data sampled from thephotodiode, and return a vector containing (time, frequency) pairs wherethe “frequency” represents maximum frequency within a short window oftime. This is accomplished through a sliding window approach with awindow length of 20 samples, for example. In one embodiment, such codewas used (“frequency_detector”). The frequency_detector eliminates allfrequencies outside the desired frequency range.

In order to test the frequency_detector code and to select the mostoptimal frequency pairs which can be detected by the Arduino®microcontroller board, simulations were run on different frequency pairsranging from 10 to 400 Hz. To enhance the accuracy of frequencyprediction, both random and 60 Hz noise were added. The most optimalfrequencies were chosen based on RMS distance calculations betweennormalized “ground truth” and result vectors. Further description ofthese embodiments of the invention is found in: Oguz Yetkin et al., Anextremely lightweight fingernail worn prosthetic interface device,University of Texas Arlington, which is hereby incorporated herein byreference in its entirety. A copy of this paper is included as Appendix“H in the provisional application to which the instant applicationclaims priority. The instant inventors are co-authors of this paper.

After reading the description presented herein, it will become apparentto a person skilled in the relevant arts how to implement embodimentsdisclosed herein using computer systems/architectures and communicationnetworks other than those described herein. It will also be appreciatedby those skilled in the relevant arts that various conventional andsuitable materials and components may be used to implement theembodiments of the invention disclosed herein.

In light of the principles and example embodiments described andillustrated herein, it will be recognized that the example embodimentscan be modified in arrangement and detail without departing from suchprinciples. Also, the foregoing discussion has focused on particularembodiments, but other configurations are also contemplated. Inparticular, even though expressions such as “in one embodiment,” “inanother embodiment,” or the like are used herein, these phrases aremeant to generally reference embodiment possibilities, and are notintended to limit the invention to particular embodiment configurations.As used herein, these terms may reference the same or differentembodiments that are combinable into other embodiments. As a rule, anyembodiment referenced herein is freely combinable with any one or moreof the other embodiments referenced herein, and any number of featuresof different embodiments are combinable with one another, unlessindicated otherwise or so dictated by the description herein. Thisdisclosure may include descriptions of various benefits and advantagesthat may be provided by various embodiments. One, some, all, ordifferent benefits or advantages may be provided by differentembodiments.

Similarly, although example methods or processes have been describedwith regard to particular steps or operations performed in a particularsequence, numerous modifications could be applied to those methods orprocesses to derive numerous alternative embodiments of the presentinvention. For example, alternative embodiments may include methods orprocesses that use fewer than all of the disclosed steps or operations,methods or processes that use additional steps or operations, andmethods or processes in which the individual steps or operationsdisclosed herein are combined, subdivided, rearranged, or otherwisealtered. Similarly, this disclosure describes one or more embodimentswherein various operations are performed by certain systems,applications, module, components, etc. In alternative embodiments,however, those operations could be performed by different components.Also, items such as applications, module, components, etc. may beimplemented as software constructs stored in a machine accessiblestorage medium, such as an optical disk, a hard disk drive, etc., andthose constructs may take the form of applications, programs,subroutines, instructions, objects, methods, classes, or any othersuitable form of control logic; such items may also be implemented asfirmware or hardware, or as any combination of software, firmware andhardware, or any combination of any two of software, firmware andhardware. The term “processor” or “microprocessor” may refer to one ormore processors.

Further, the methods set forth herein may also be implemented as anarticle of manufacture embodiment, wherein an article of manufacturecomprises a non-transitory machine-accessible medium containinginstructions, the instructions comprising a software application orsoftware service, wherein the instructions, when executed by themachine, cause the machine to perform the respective method. The machinemay be, e.g., a processor, a processor-based system such as the systemsdescribed herein, or a processor-based device such as the user interfacedevices described herein.

In view of the wide variety of useful permutations that may be readilyderived from the example embodiments described herein, this detaileddescription is intended to be illustrative only, and should not be takenas limiting the scope of the invention. What is claimed as theinvention, therefore, are all implementations that come within the scopeof the following claims, and all equivalents to such implementations.

What is claimed is:
 1. A system, comprising: a signal element configuredto generate a signal; a fingernail mount configured to secure the signalelement on a fingernail of a first finger of a user; and a detectionelement configured for disposal on a second finger of a user, andconfigured to detect a signal generated by the signal element, whereinthe signal indicates that a gesture has been performed using the firstfinger and the second finger of the user.
 2. The system according toclaim 1, wherein the detection element is further configured to detect acharacteristic of the signal, the characteristic of the signalindicative of a characteristic of the gesture.
 3. The system accordingto claim 2, wherein the characteristic of the signal comprises an RFIDtag ID, and wherein the detection element is configured to distinguishthe first finger from a plurality of other fingers based on the RIFD tagID.
 4. The system according to claim 2, wherein the characteristic ofthe signal comprises a signal intensity, strength, phase, or amplitude,or change therein.
 5. The system according to claim 2, wherein thecharacteristic of the signal comprises a resonant frequency of a circuittransmitting or generating the signal.
 6. The system according to claim2, wherein the characteristic of the signal comprises a frequency orpulse pattern.
 7. The system according to claim 1, further comprising acontroller configured to control a prosthetic device based on the signaldetected by the detection element.
 8. The system according to claim 6,wherein the control of the prosthetic device causes the prostheticdevice to perform an action or a movement corresponding to the gesture.9. The system according to claim 1, wherein the signal element comprisesan RFID tag, and wherein the detection element comprises an RFIDantenna.
 10. The system according to claim 1, wherein the signal elementcomprises an LC circuit.
 11. The system according to claim 1, whereinthe detection element comprises a circuit configured to distinguish thesignal from the signal element from a plurality of other signals fromother signal elements disposed on a plurality of other fingers of theuser.
 12. The system according to claim 1, wherein the circuit isconfigured to distinguish the signal by determining that an intensity ofthe signal from the signal element is greater than the intensities ofthe other signal elements.
 13. The system according to claim 1, whereinthe signal element comprises a resonator and the receiver element isconfigured to distinguish the signal element based on a resonancefrequency.
 14. The system according to claim 13, wherein the resonatorcomprises an LC tank circuit.
 15. The system according to claim 13,wherein the resonator comprises an RFID circuit.
 16. The systemaccording to claim 13, wherein the resonator comprises a microcontrollercircuit configured to respond with a specified signal unique to theuser's first finger or second finger.
 17. The system according to claim1, wherein the signal indicates that a sign language gesture has beenperformed using at least the first finger and the second finger of theuser.